33 research outputs found
Overview of the CLEF 2018 Consumer Health Search Task
This paper details the collection, systems and evaluation
methods used in the CLEF 2018 eHealth Evaluation Lab, Consumer Health Search (CHS) task (Task 3). This task investigates the effectiveness of search engines in providing access to medical information present on the Web for people that have no or little medical knowledge. The task aims to foster advances in the development of search technologies for Consumer Health Search by providing resources and evaluation methods to test and validate search systems. Built upon the the 2013-17 series of CLEF eHealth Information Retrieval tasks, the 2018 task considers
both mono- and multilingual retrieval, embracing the Text REtrieval Conference (TREC) -style evaluation process with a shared collection of documents and queries, the contribution of runs from participants and the subsequent formation of relevance assessments and evaluation of the participants submissions.
For this year, the CHS task uses a new Web corpus and a new set of queries compared to the previous years. The new corpus consists of Web pages acquired from the CommonCrawl and the new set of queries consists of 50 queries issued by the general public to the Health on the Net (HON) search services. We then manually translated the 50 queries to
French, German, and Czech; and obtained English query variations of the 50 original queries.
A total of 7 teams from 7 different countries participated in the 2018 CHS task: CUNI (Czech Republic), IMS Unipd (Italy), MIRACL (Tunisia), QUT (Australia), SINAI (Spain), UB-Botswana (Botswana), and UEvora (Portugal)
Overview of the CLEF 2018 Consumer Health Search Task
This paper details the collection, systems and evaluation
methods used in the CLEF 2018 eHealth Evaluation Lab, Consumer
Health Search (CHS) task (Task 3). This task investigates the effectiveness of search engines in providing access to medical information present
on the Web for people that have no or little medical knowledge. The task
aims to foster advances in the development of search technologies for
Consumer Health Search by providing resources and evaluation methods
to test and validate search systems. Built upon the the 2013-17 series
of CLEF eHealth Information Retrieval tasks, the 2018 task considers
both mono- and multilingual retrieval, embracing the Text REtrieval
Conference (TREC) -style evaluation process with a shared collection of
documents and queries, the contribution of runs from participants and
the subsequent formation of relevance assessments and evaluation of the
participants submissions.
For this year, the CHS task uses a new Web corpus and a new set of
queries compared to the previous years. The new corpus consists of Web
pages acquired from the CommonCrawl and the new set of queries consists of 50 queries issued by the general public to the Health on the Net
(HON) search services. We then manually translated the 50 queries to
French, German, and Czech; and obtained English query variations of
the 50 original queries.
A total of 7 teams from 7 different countries participated in the 2018 CHS
task: CUNI (Czech Republic), IMS Unipd (Italy), MIRACL (Tunisia),
QUT (Australia), SINAI (Spain), UB-Botswana (Botswana), and UEvora
(Portugal)
ShARe/CLEF eHealth evaluation lab 2014, task 3: user-centred health information retrieval
This paper presents the results of task 3 of the ShARe/CLEF eHealth Evaluation Lab 2014. This evaluation lab focuses on improving access to medical information on the web. The task objective was to investigate the effect of using additional information such as a related discharge summary and external resources such as medical ontologies on the IR effectiveness, in a monolingual and in a multilingual context. The participants were allowed to submit up to seven runs for each language, one mandatory run using no additional information or external resources, and three each using or not using discharge summaries
Recommended from our members
The future of sleep health: a data-driven revolution in sleep science and medicine.
In recent years, there has been a significant expansion in the development and use of multi-modal sensors and technologies to monitor physical activity, sleep and circadian rhythms. These developments make accurate sleep monitoring at scale a possibility for the first time. Vast amounts of multi-sensor data are being generated with potential applications ranging from large-scale epidemiological research linking sleep patterns to disease, to wellness applications, including the sleep coaching of individuals with chronic conditions. However, in order to realise the full potential of these technologies for individuals, medicine and research, several significant challenges must be overcome. There are important outstanding questions regarding performance evaluation, as well as data storage, curation, processing, integration, modelling and interpretation. Here, we leverage expertise across neuroscience, clinical medicine, bioengineering, electrical engineering, epidemiology, computer science, mHealth and human-computer interaction to discuss the digitisation of sleep from a inter-disciplinary perspective. We introduce the state-of-the-art in sleep-monitoring technologies, and discuss the opportunities and challenges from data acquisition to the eventual application of insights in clinical and consumer settings. Further, we explore the strengths and limitations of current and emerging sensing methods with a particular focus on novel data-driven technologies, such as Artificial Intelligence
CLEF eHealth 2019 Evaluation Lab
Since 2012 CLEF eHealth has focused on evaluation resource building efforts around the easing and support of patients, their next-of-kins, clinical staff, and health scientists in understanding, accessing, and authoring eHealth information in a multilingual setting. This year’s lab offers three tasks: Task 1 on multilingual information extraction; Task 2 on technology assisted reviews in empirical medicine; and Task 3 on consumer health search in mono- and multilingual settings. Herein, we describe the CLEF eHealth evaluation series to-date and then present the 2019 tasks, evaluation methodology, and resources
Overview of the CLEF eHealth Evaluation Lab 2019
In this paper, we provide an overview of the seventh annual
edition of the CLEF eHealth evaluation lab. CLEF eHealth 2019 continues our evaluation resource building efforts around the easing and support of patients, their next-of-kins, clinical staff, and health scientists in
understanding, accessing, and authoring electronic health information in
a multilingual setting. This year’s lab advertised three tasks: Task 1 on
indexing non-technical summaries of German animal experiments with
International Classification of Diseases, Version 10 codes; Task 2 on technology assisted reviews in empirical medicine building on 2017 and 2018
tasks in English; and Task 3 on consumer health search in mono- and
multilingual settings that builds on the 2013–18 Information Retrieval
tasks. In total nine teams took part in these tasks (six in Task 1 and three in Task 2). Herein, we describe the resources created for these tasks and
evaluation methodology adopted. We also provide a brief summary of
participants of this year’s challenges and results obtained. As in previous years, the organizers have made data and tools associated with the
lab tasks available for future research and development
Overview of the CLEF eHealth Evaluation Lab 2018
In this paper, we provide an overview of the sixth annual edition of the CLEF eHealth evaluation lab. CLEF eHealth 2018 continues
our evaluation resource building efforts around the easing and support of
patients, their next-of-kins, clinical staff, and health scientists in understanding, accessing, and authoring eHealth information in a multilingual
setting. This year’s lab offered three tasks: Task 1 on multilingual information extraction to extend from last year’s task on French and English
corpora to French, Hungarian, and Italian; Task 2 on technologically
assisted reviews in empirical medicine building on last year’s pilot task in English; and Task 3 on Consumer Health Search (CHS) in mono- and
multilingual settings that builds on the 2013–17 Information Retrieval
tasks. In total 28 teams took part in these tasks (14 in Task 1, 7 in Task
2 and 7 in Task 3). Herein, we describe the resources created for these
tasks, outline our evaluation methodology adopted and provide a brief
summary of participants of this year’s challenges and results obtained.
As in previous years, the organizers have made data and tools associated
with the lab tasks available for future research and development
Diagnose this if you can: On the effectiveness of search engines in finding medical self-diagnosis information
An increasing amount of people seek health advice on the web using search engines; this poses challenging problems for current search technologies. In this paper we report an initial study of the effectiveness of current search engines in retrieving relevant information for diagnostic medical circumlocutory queries, i.e., queries that are issued by people seeking information about their health condition using a description of the symptoms they observes (e.g. hives all over body) rather than the medical term (e.g. urticaria). This type of queries frequently happens when people are unfamiliar with a domain or language and they are common among health information seekers attempting to self-diagnose or self-treat themselves. Our analysis reveals that current search engines are not equipped to effectively satisfy such information needs; this can have potential harmful outcomes on people’s health. Our results advocate for more research in developing information retrieval methods to support such complex information needs
Consumer health search on the web: study of web page understandability and its integration in ranking algorithms
Understandability plays a key role in ensuring that people accessing health information are capable of gaining insights that can assist them with their health concerns and choices. The access to unclear or misleading information has been shown to negatively impact the health decisions of the general public.The aim of this study was to investigate methods to estimate the understandability of health Web pages and use these to improve the retrieval of information for people seeking health advice on the Web.Our investigation considered methods to automatically estimate the understandability of health information in Web pages, and it provided a thorough evaluation of these methods using human assessments as well as an analysis of preprocessing factors affecting understandability estimations and associated pitfalls. Furthermore, lessons learned for estimating Web page understandability were applied to the construction of retrieval methods, with specific attention to retrieving information understandable by the general public.We found that machine learning techniques were more suitable to estimate health Web page understandability than traditional readability formulae, which are often used as guidelines and benchmark by health information providers on the Web (larger difference found for Pearson correlation of .602 using gradient boosting regressor compared with .438 using Simple Measure of Gobbledygook Index with the Conference and Labs of the Evaluation Forum eHealth 2015 collection).The findings reported in this paper are important for specialized search services tailored to support the general public in seeking health advice on the Web, as they document and empirically validate state-of-the-art techniques and settings for this domain application
MM: a new framework for multidimensional evaluation of search engines
In this paper, we proposed a framework to evaluate information retrieval systems in presence of multidimensional relevance. This is an important problem in tasks such as consumer health search, where the understandability and trustworthiness of information greatly influence people's decisions based on the search engine results, but common topicality-only evaluation measures ignore these aspects. We used synthetic and real data to compare our proposed framework, named MM, to the understandability-biased information evaluation (UBIRE), an existing framework used in the context of consumer health search. We showed how the proposed approach diverges from the UBIRE framework, and how MM can be used to better understand the trade-offs between topical relevance and the other relevance dimensions